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ABSTRACT 



This paper critically reflects upon the questions asked and 
the models and methods employed to evaluate online learning in several 
studies. These online studies range from setting up online networks for 
social interaction and facilitating collaborative learning experiences among 
graduate students in several universities, to moderating worldwide online 
professional development activities. The paper examines the methodologies 
used to answer the following evaluation research questions: (1) How can we 

describe online participation, interaction patterns, and group dynamics? (2) 
Were students satisfied with their online learning experience? (3) Did 
participants learn? and (4) Was knowledge constructed? It is concluded that 
the adoption of a single technique for analyzing the quality of the learning 
experience in online learning networks has not yielded satisfactory answers, 
and a mixed methodology approach is recommended to study the complex nature 
of online learning networks. (Contains 11 references.) (MES) 
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Abstract: 

This paper critically reflects upon the questions we have asked and the models and 
mefliods we have enq>loyed to evaluate o nline learning in several stuifies. The paper 
examines the methodologies used to answer the following evaluation research questions; 
1. How can we describe online participation, interaction patterns, and group dynamics? 2. 
Were students satisfied with the experience of participafing in the conference? 3. Did 
participants learn? 4. Was knowledge constructed? 



One of the critical challenges facing educators as we move into the twenty-first century is 
the development of ^ipropriate methodologies and tools for evaluating online learning 
networks. The open-ended nature of online learning projects, the multiple threads of 
conversation, and the fluid participation patterns pose new challenges to evaluators. 
Onlin e learning designs are often based on constructivist, learner- centered principles 
which provide more learner control, facilitate the sharing of multiple perspectives, and 
places emphasis on individual learners creating their own me aning . Behavioral 
objectives with a stated outcome for all learners, is not the goal of many online learning 
projects. Traditional evaluation methods used to evaluate learning within the four walls 
of a classroom do not transfer well to the online context Evaluators therefore, are 
challenged to imderstand the imique characteristics of the online medium and its social 
and ecological structure in order to develop new principles for evaluating learning. 

The purpose of this paper is to critically reflect upon the questions we have asked and the 
models and methods we have employed to evaluate online learning in several studies we 
have conducted. These online studies range from setting up online networks for social 
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interaction, facilitating collaborative learning experiences among graduate students in 
several universities, to moderating worldwide o nlin e professional development activities. 

This paper will be organized according to the evaluation questions we have asked and 
attempted to answer: 

1. How can we describe online participation, interaction patterns, and group dynamics? 

2. Were students satisfied with the experience of participating in the conference? Did 
they feel that it had been a worthwhile use of their time, and one that they would be 
willing to repeat? 

These questions were answered using a variety of techniques; both quantitative and 
qualitative. However, while these questions have value in understanding the nature of a 
computer conference (CC) , they do not yield useful information on the quality of the 
learning that took place. 

One significant question which has yet to be satisfactorily answered is: How do you 
assess the quality of the learning experience in a computer-mediated conference (CMC)? 
Questions we need to ask are: 

3. Did participants leam? 

4. Was knowledge constructed? 

Currently, the question that has intrigued us, is: How is knowledge constructed in 
online learning networks through the process of social negotiation? In order to address 
this challenging question, content analysis or interaction analysis of computer transcripts 
using qualitative research techniques is essential. We have relied on interaction analysis 
in our attempts to develop an interaction analysis model for examining social 
construction of knowledge in online learning networks (Gunawardena, Lowe, and 
Anderson, 1997) which we will describe in this paper. 



How Can We Describe Online Participation, Interaction Patterns, and Group Dynamics? 

Participation analysis techniques examine the capacity of a conference to engage 
members and reveal comparative patterns of participation among learners from varying 
backgrormds. An evaluation tool we have employed to address these questions is the 
model developed by Levin, Kim and Riel (1990) for analyzing instructional interactions 
on electronic message networks. This model has four dimensions of analysis: 1) 
Participant Structures Analysis, 2) Intermessage Reference Analysis, 3) Message Act 
Analysis, and 4) Message Flow analysis. We used this model to evaluate a peer support 
network for medical students at the University of New Mexico (UNM), set up to maintain 
the social and psychological siq>port network during the period of time that they worked 
with preceptors in isolated rural communities in New Mexico (Gunawardena, Gittinger, 
and Eivorak, 1991). 

In reflecting on the use of the Levin, et al. (1990) model, we fotmd that while it is a good 
starting point for obtaining an overall picture of the network group, activities, and 
interaction, it did not fit neatly with the purpose of our project. The Intermessage 
Reference Analysis was difficult to apply since our application of CMC was primarily 




social in nature. AH participants on-line were peers; no instructor activity existed. The 
purpose of these conununications was not academic and tasks were not performed on- 
line. As a result, information such as whose messages were referenced most, what topics 
were referenced most, did not stand out in any particular way. Levin, et al. do not 
address content analysis within Message Act Analysis. We found our research was 
enriched by a careful analysis of the content of the interaction. We have foimd other data 
such as unsolicited participant reactions online, solicited participant reactions, both 
instructor and student perspectives on interaction and the analysis of computer transcripts 
to be more useful in forming a picture of events that occurred in the online community 
(Gxmawardena, 1993). 



Were Learners Satisfied with Their Online Learning Experiences? 

The methods we have used most often to solicit learner reactions to and satisfaction with 
online learning exp^ences include online surveys, paper and pencil surveys, and 
participant reactions to the conference posted online in a conference area designed for 
this purpose. Another important source of data are the unsolicited comments on 
participant satisfaction or dissatisfaction with the learning experience that occur 
throughout the transcript. 

Structured survey questions are useful in obtaining an overall view of student reactions to 
the conference and their satisfaction. It is also possible to obtain more in-depth data on 
selected open-ended questions. The survey is an easy vehicle to obtain data when 
students are scattered across geographic distance. Analysis of survey data has taken many 
forms depending on the questions we have asked related to various studies we have 
conducted. 

Hiltz ( 1990) discussed an approach to determining learner satisfaction by examining the 
social psychological (characteristics of the users); human relations (characteristics of the 
groups and organizations within which systems are implemented); and technological 
determinist (characteristics of the system); factors that impact student satisfaction with, 
and subsequent use of computer conferencing. Using her theoretical fiamework we 
examined a computer conference in which a group of graduate students fi’om the 
University of New Mexico joined 13 academic experts firom 7 coimtries to discuss 
“cooperative learning” within the' context of a global electronic pre-conference for the 
international TeleTeaching '93 conference held in Norway (Gunawardena and Heeren, 
1993). The analysis was based on a questioimaire diat addressed Hiltz’s dimensions. One 
of the major problems associated widi our study and with quantitative analysis when 
applied to the CMC context was sample size and sample selection. While multiple 
regression and analysis of variance were conducted to examine these variables, the 
statistical significant differences are not convincing because of the small number of 
participants in our study. 

A recent quantitative study we conducted to answer the question: What are the variables 
that can predict student satisfaction in online learning networks? yielded more positive 




outcomes (Gunawardena & Zittle, 1997). The purpose of this research was to examine 
which process variables such as proficiency in technical skills, learner support, and 
“social presence” or the degree to which a person is perceived as “real” in mediated 
communication (Short, Williams, and Christie, 1976), could predict learner satisfaction 
within a CMC environment. The process and satisfaction variables were measured by a 
61 -item questionnaire consisting of predominantly five-point Likert scale items specially 
designed for this study. 

The subjects were fifty graduate students from five universities: San Diego State (N=8), 
Texas A «& M (N=l 1), University of New Mexico (N=14), University of Wisconsin- 
Madison (N=7), and University of Wyoming (N=10), who participated in the Fall 1993 
Globaled inter-university virtual conference. The Globaled conference was an academic 
exercise that was a class requirement at each participating university (Gunawardena, et. 
al. 1994). 

In this analysis (Gunawardena and Zittle, 1997), eight process variables were used to 
predict overaU satisfaction with the Globaled conference. The stepwise regression 
analysis converged on a three-predictor model revealing that social presence, student 
perception of having equal opportunity to participate in the conference, and proficiency 
in technical skills accounted for about 68% of the explained variance. Social presence 
alone contributed about 60% of this variance, suggesting that it may be a very strong 
predictor of satisfaction. Although care must be taken in generalizing these results due to 
sample selection and size, the results suggest that social presence alone is a strong 
predictor of satisfection in a text-based computer conference. 

When reflecting on the quantitative analyses we have done using surveys, we feel that the 
strengths of these analyses lie their ability to point out salient differences in process 
variables as they relate to learner satisfaction. One of the shortcomings of quantitative 
analysis however, is the inability to explain the reasons for the significant differences 
observed. This is where qualitative analysis of open-ended questions or interviews is of 
value. A related problem with quantitative analysis when applied to the CMC context, is 
the small sample size in many conferences, and problems related to sample selection as 
random sampling is difficult in the distance education context. Quantitative analyses 
should be p^ormed only when there is an adequate sample size to study the variables of 
interest. Qualitative data used in conjunction with quantitative data can usually e?q>lain 
the significant differences found. Therefore, we would advocate a mixed methodology 
approach (Tashakkori & Teddlie, 1998) combining qxiantitative and qualitative data to 
overcome some of the problems related to quantitative data alone. 



Did participants learn? 

How did the individual learner make sense of the computer conference? What did he or 
she leam? These questions are more difficult to answer if we subscribe to a constructivist 
view of learning where the individual learner is expected to take away a unique 
perspective from the learning experience. Traditional methods of measuring learning 




using pre and post tests will not work well in this context as learning occurs in many 
different forms and is not limited to learning the content or subject imder study. We 
have been interested in ex amining questions related to learning from two perspectives: 
What did participants learn about the subject/topic/content that was discussed? and What 
did they learn about the medium of computer conferencing and it’s influence on the 
learning process? 

One method is to analy 2 e the computer transcript as it affords an unobtrusive, and fairly 
accurate means of gauging whether participants learned during the conference. If one 
looks carefully at the transcript, we can decipher several unsolicited comments or 
“ thinking aloud” comments that refer to individual learning such as the following in a 
debate that discussed the importance of ‘interaction’ to distance education: “In the past 
two days, you who are participating in this conversation have made me stop and think 
about ‘interaction.’ I guess you’d call that ‘learning.’ Without your thoughts this would 
not have happened. I think this demonstrates the importance of ‘interaction’ to learning.” 
(Gunawardena, Lowe, and Anderson, 1995, p. 202). 

Another technique is to ask participants directly what they learned, either through open- 
ended questions in questionnaires, or individual interviews, or, asking them to discuss 
their learning in a separate conference space specially designed for this purpose. In a 
study where we used open-ended questions in a questionnaire to find out what students 
learned about “cooperative learning,” the topic of the conference, we found that students 
learned about: 1. cooperative learning as an educational concept, 2. how to participate in 
a conference that generated diverse opinions, 3. what makes a good and a bad conference, 
and 4. how to use the technology to participate in a computer conference (Gunawardena 
and Heeren, 1993). 

Evaluation data on student learning can also be collected online by asking very general 
questions such as: What did you think about this computer conferencing experience? 

In order to imderstand the myriad forms of learning that occurs in a computer conference, 
we have often asked students to keep weekly journals documenting all aspects of 
learning. These journals have given us a unique perspective of each individual’s learning 
process. Other techniques we have used are to ask students to critique their online 
learning experiences and to apply and transfer what they have learned finrn the computer 
conference to developing a computer conferencing design. 

In reflecting on the various evaluation techniques we have used to imderstand student 
learning, we have formd that qualitative approaches work better than quantitative 
methods in being able to understand the unique ways in which individual students learn. 



Was Knowledge Constructed? 

Perhaps the most challenging and the most exciting question one can ask in evaluating 
online learning is: Was knowledge constructed within the group by means of the 




exchanges among participants? We will discuss two interaction analysis models we have 
used to answer this question. Initially, we began to use Henri’s (1992) model to analyze 
the transcripts of a global online debate we had conducted as an adult professional 
development experience. However, it became clear that three aspects of Henri's model; its 
basis in a teacher-centered instructional paradigm, its distinction between the cognitive 
and the metacognitive dimensions, and its treatment of the concept of interaction, were 
unsuited for application to the debate. We felt that we needed to develop a new 
definition of interaction for the CMC context if we are to examine the process of social 
negotiation (Gunawardena, Lowe, and Anderson, 1997). 

We believe that the metaphor of a patchwork quilt better describes the process of shared 
construction of knowledge that occurs in a constructivist learning environment. A quilt 
block is built up by the application, one after another, of small pieces of cloth, which 
when assembled form a bright and colorful pattern. The pieces, according to this analogy, 
are the contributions of individual participants. Each participant contributes to the whole 
his or her own texture and color of thought, jxist as every scrap of fabric forms a 
distinctive element in the overall pattern. TTie pattern may not be complete during a 
single conference, but individual responses can contribute toward the formation of a 
pattern. The process by which the contributions are fitted together is interaction, broadly 
understood, and the pattern which emerges at the end, when the entire gestalt of 
accumulated interaction is viewed, is the newly-created knowledge or meaning. 
Interaction is the essential process of putting together the pieces in the co-creation of 
knowledge. 

Based on this new definition of interaction, and grounded theory principles, we 
developed an outline of the process of negotiation which appears to occur in the co- 
construction of knowledge. The outline led to the development of the interaction analysis 
model which has five phases, reflecting the complete process of negotiation which must 
occur when there are substantial areas of inconsistency or disagreement to be resolved. 
The phases of learning outlined in this model occur at both the individual and social level 
and can be described as: 

Phase I: Sharing/Comparing, 

Phase II: Dissonance, 

Phase HI: Negotiation/Co-construction, 

Phase IV: Testing Tentative Constructions, and 

Phase V: Statement/Application of Newly-Constructed Knowledge. 

A detailed discussion of this model and its underlying theoretical finmeworic is found in 
Gunawardena, Lowe, and Anderson (1997). 

Two major themes were observed related to knowledge construction. One was the 
progress of certain strands of argument firom Phase I to Phase V which can be described 
as an exercise in the co-construction of knowledge, moving firom lower to higher mental 
functions. The other was the evidence of more than one and sometimes three phases 
within a single message posted by one participant, which usually progressed in sequence 
through the phases, showing progress fi'om lower to higher mental functions, showing 
how individuals contributed toward the co-construction. 




Conclusion 



The adoption of a single technique for analyzing the qiiality of the learning experience in 
online learning networks has not yielded satisfactory answers. The complex nature of 
online learning calls for the use of multiple methods and multiple soinces of data to 
understand group as well as individual learning. While quantitative analysis of interaction 
patterns and learner satisfaction has yielded useful information on the overall nature of 
online learning and the variables that influence its success, several problems exist in the 
application of the quantitative paradigm to evaluating learning in the o nlin e context. Two 
problems relate to sample size and selection. While quantitative data can point out a 
significant difference, it cannot adequately explain the reasons for the observed 
difference. The naturalistic paradigm employing qualitative data fi-om in-depth 
interviews, observations, and computer transcript analysis, can often explain the observed 
differences. When used in conjtmction with quantitative data, qualitative data can 
overcome some of the shortcomings of utilizing quantitative data alone. Therefore, it is 
recommended that a mixed methodology approach be used to study the complex nature 
of online learning networks. 



References: 

Gunawardena, C. N., Campbell Gibson, C., Cochenour, J. J., Dean, T., Dillon, C. L., HessmilleT, R,, Mmphy, K., 
Rezabek, L. L., Saba, F. (1994). Multiple perspectives on implementing inter-univetsity computer conferencing. 
Pmceedings of the Distance lisaTninp Rfyaanch Conference (pp. 101-117). College Station, TX: Texas A&M 
University, Dept of Educational Human Resource Development 

Gunawardena, C. N., Gittinger Jr., J. D., & Dvorak, L. P. (1991). The design, imi^ementation and evaluation of a 
computer-mediated communication support system for medical students. Proceedings of the Seventh Annual 
Conference on Distance Teaching and l.eaming (pp. 98-102). Madison, WI: University of Wisconsin-Madison. 

Gunawardena, C. N., & Heeren, E. (1993). Design and analysis of global cooperative learning through conqjuter 
conferencing. Proceedings of the Ninth Annual Conference on Di-stance Teaching and I .earning Inn <i0-73\ 
Madison, WI: University of Wisconsin-Madison. 



Gunawardena, C. N., Lowe, C., & Airdetson, T. (1995). The design, implementation and evaluation of a worldwide 
online debate. Proceedings of the International Conference on Technology and Distance Bdncation (VI Congreso 
Intemadonal: Technologia y Educacion A Distancia) (pp.195-206), San Jose, Costa Rica: UNED Press. 

Gunawardena, C. N., Lowe, C. A., & Anderson, T. (1997). Analysis of a global online debate and the development of 
an interaction analysis model for e xamining social construction of knowledge in computer confereixnng. Journal of 
Educational Comriuting Research. 17 (41. 395-429. 

Gunawardena, C. N., & Zhtle, F. (1997). Social presence as a predictor of satisfection within a conqiuter mediated 
conferencing environment American Journal of Distarrce Education. 1 I Gl. 8-25. 

Henri, F. (1992). Conqniter coiferencing and content analysis. In A. R. Kaye (Ed.), Collaborative learning through 
computer conferencing: The Naiaden nanets (pp. 1 17-136). Berlin, New York: Springer-Verlag. 

Hiltz, S. R. (1990). Evaluating the Virtual Classroom. In LAI. Harasim (Ed.), Online Pdncation (133-183). New York: 
Praeger. 



BEST COPY AVAILABLE 



Levin, J. A., Kim, R, & Riel, M. M. (1990). Analyzing instructional interactions on electronic message networks. In 
L. Haiasim (Ed.), Online education f 18S-213V New York; Praeger. 



Short, J., Williams, E., & Christie, B. (1976). The social psychology of telecommunications . Lorrdon; John Wiley & 
Sons. 



Tashakkori, A., & Teddlie, C. (1998). Mixed methodoloev: Oomhining qualitative and quantitative atmroaches . 
Thousand Oaks, CA: Sage. 



U.S. Department of Education 

Office of Educational Research and Improvement (OERI) 
National Library of Education (NLE) 
Educational Resources Information Center (ERIC) 





NOTICE 

REPRODUCTION BASIS 



This document is covered by a signed “Reproduction Release 
(Blanket) form (on file within the ERIC system), encompassing all 
or classes of documents from its source organization and, therefore, 
does not require a “Specific Document” Release form. 



This document is Federally-funded, or carries its own permission to 
reproduce, or is otherwise in the public domain and, therefore, may 
be reproduced by ERIC without a signed Reproduction Release form 
(either “Specific Document” or “Blanket”). 





